python, cluster computing, design help [closed]

Posted by j dawg on Programmers See other posts from Programmers or by j dawg
Published on 2011-11-13T02:55:17Z Indexed on 2011/11/21 10:19 UTC
Read the original article Hit count: 523

Filed under:

python

|

server

|

cluster

I would like to create my own parallel computing server. Can you please point me to some resources I can use to help me develop my server.

Sorry, like I said I need help getting started.

Yes, I am limited to python, I cannot use C. I am using a bunch of workstations and I want to use all the cpus in those machines.

So what I am looking for is blog posts, books, articles that can help me develop my own client/server tools to send code from the client to the servers and spawn python processes based on the number of cpus.

I know how to do the subprocessing/multiprocessing part of the program, I do not know how to create the server that will take the client's requests. I also need to figure out what is the best way to handle sending file data, like netcdf files or other spatial data.

Any suggestions very welcome.

© Programmers or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about server

SQL SERVER – Server Side Paging in SQL Server 2011 Performance Comparison

as seen on SQL Authority - Search for 'SQL Authority'
Earlier, I have written about SQL SERVER – Server Side Paging in SQL Server 2011 – A Better Alternative. I got many emails asking for performance analysis of paging. Here is the quick analysis of it. The real challenge of paging is all the unnecessary IO reads from the database. Network traffic was… >>> More
Should I switch my server to Ubuntu Server from Windows Server 2003

as seen on Server Fault - Search for 'Server Fault'
I have a server thats primarily used for SRCDS, I host a few Team Fortress 2 servers on it and I would much prefer a ssh-like (more so the no-gui part) for something like this, mostly because I can't navigate with my phone on VNC very well and also because there really is no need for a gui in this… >>> More
Windows web server and SQL Server on same dedicated server

as seen on Server Fault - Search for 'Server Fault'
I'm currently trying to decide on the best approach to handle hosting a few moderate traffic websites for production e-commerce and online applications. We'd like to move to a dedicated server and are looking at this as the most likely machine: Quad Core Intel Core2Quad Q9550 Processor, 2.83 Ghz… >>> More
Windows 7 Desktop/Start Menu Redirection: Server O/S: Windows Server 2003 And Server 2008

as seen on Super User - Search for 'Super User'
Hi, I am new here so I am might be asking a question which has already been answered [however I can't see it in the suggested answers above] I manage a network which is split into a parent domain and a child domain. Recently I have been looking at when to migrate to Windows 7. The child domain… >>> More
Windows 7 Desktop/Start Menu Redirection: Server O/S: Windows Server 2003 And Server 2008

as seen on Server Fault - Search for 'Server Fault'
Hi, I am new here so I am might be asking a question which has already been answered [however I can't see it in the suggested answers above] I manage a network which is split into a parent domain and a child domain. Recently I have been looking at when to migrate to Windows 7. The child domain… >>> More